Engineering EE 194 – Network Information Theory Prof . Mai Vu Lecture 2 : Channel Capacity 1 Typicality and Joint Typicality
نویسنده
چکیده
The Asymptotic Equipartition Property (AEP) in information theory is the analog of the Law of Large Numbers in probability theory. Assume that the binary random variable X has a probability mass function defined by p(1) = 2/3, and p(0) = 1/3, where X1,X2,...,Xn are i.i.d. random variables according to p(x), and we want to observe a realization of the sequenceX1,X2,...,Xn, where n −→ ∞. We will see that as n increases, the number of zeroes and ones in the sequence would be very close to n/3, and 2n/3, respectively.
منابع مشابه
Quantum Information Chapter 10. Quantum Shannon Theory
Contents 10 Quantum Shannon Theory 1 10.1 Shannon for Dummies 2 10.1.1 Shannon entropy and data compression 2 10.1.2 Joint typicality, conditional entropy, and mutual information 6 10.1.3 Distributed source coding 8 10.1.4 The noisy channel coding theorem 9 10.2 Von Neumann Entropy 16 10.2.1 Mathematical properties of H(ρ) 18 10.2.2 Mixing, measurement, and entropy 20 10.2.3 Strong subadditivit...
متن کاملA Joint Typicality Approach to Algebraic Network Information Theory
This paper presents a joint typicality framework for encoding and decoding nested linear codes for multi-user networks. This framework provides a new perspective on compute–forward within the context of discrete memoryless networks. In particular, it establishes an achievable rate region for computing the weighted sum of nested linear codewords over a discrete memoryless multiple-access channel...
متن کاملTypicality Matching for Pairs of Correlated Graphs
In this paper, the problem of matching pairs of correlated random graphs with multi-valued edge attributes is considered. Graph matching problems of this nature arise in several settings of practical interest including social network deanonymization, study of biological data, web graphs, etc. An achievable region for successful matching is derived by analyzing a new matching algorithm that we r...
متن کاملVariable-Rate Coding with Feedback for Universal Communication Systems
Classical coding schemes that rely on joint typicality (such as Slepian-Wolf coding and channel coding) assume known statistics and rely on asymptotically long sequences. However, in practice the statistics are unknown, and the sequences are of finite length n. In this finite regime, we must allow a non-zero probability of coding error and also pay a penalty by transmitting extra bits. The pena...
متن کاملTypical Sequences for Polish Alphabets
The notion of typical sequences plays a key role in the theory of information. Central to the idea of typicality is that a sequence x1, x2, . . . , xn that is PX -typical should, loosely speaking, have an empirical distribution that is in some sense close to the distribution PX . The two most common notions of typicality are that of strong (letter) typicality and weak (entropy) typicality. Whil...
متن کامل